Measurement and Analysis of TCP Throughput Collapse in Cluster-based Storage Systems

Cluster-based and iSCSI-based storage systems rely on standard TCP/IP-over-Ethernet for client access to data. Unfortunately, when data is striped over multiple networked storage nodes, a client can experience a TCP throughput collapse that results in much lower read bandwidth than should be provided by the available network links. Conceptually, this problem arises because the client simultaneously reads fragments of a data block from multiple sources that together send enough data to overload the switch buffers on the client’s link. This paper analyzes this Incast problem, explores its sensitivity to various system parameters, and examines the effectiveness of alternative TCPand Ethernet-level strategies in mitigating the TCP throughput collapse. Acknowledgements: We would like to thank Jeff Butler, Abbie Matthews, and Brian Mueller at Panasas Inc. for allowing us and helping us to conduct experiments on their systems. We thank the members and companies of the PDL Consortium (including APC, Cisco, EMC, Google, Hewlett-Packard, Hitachi, IBM, Intel, LSI, Network Appliance, Oracle, Seagate, and Symantec) for their interest, insights, feedback, and support. Finally, we’d like to thank Michael Stroucken for his help managing the PDL cluster, and Michael Abd-el-Malek for feedback on our work. This material is based on research sponsored in part by the National Science Foundation, via grants #CNS-0546551, #CNS-0326453 and #CCF-0621499, by the Army Research Office under agreement number DAAD19–02–1–0389, by the Department of Energy under Award Number #DE-FC0206ER25767, and by DARPA under grant #HR00110710025.